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ABSTRACT 

Recently, several industrialized nations have engaged 
in extensive and expensive assessments of adult literacy using 
door-to-door surveys and tests of "functional" reading. A study 
examined the use of simple checklists of author's names, magazine 
titles, famous people's names, and vocabulary works used earlier by 
West, Stanovich and Miller (1993) to sample adults 1 declarative 
knowledge by telephone. Data were derived from telephone interviews 
with 538 randomly selected adults residing in the San Diego, 
California, metropolitan area, and by a survey mailed to a subsample 
of 280 of these adults who agreed to participate (abou^ 50 percent of 
the surveys were returned). Analyses revealed strong relationships 
among checklist knowledge (r = .80) when assessed either by listening 
(telephone) or by reading (mailed surveys) . A comparison of findings 
using the telephone methodology with that of the recent U.S. 
Department of Education's National Adult Literacy Survey (h iLS) 
showed that all major relationships of literacy to education, age, 
gender, occupation, income, ethnicity, father's and mother's 
education that have been found in the NALS and other literacy surveys 
over the past 75 years were found with the telephone method. Given 
the much lower cost of telephone versus door-to-door surveys (less 
than ore-sixth the cost), the results suggest that the assessment of 
adult literacy by telephone is a promising and cost-effective method 
that should be further developed as a means of monitoring national 
progress toward the achievement of greater levels of adult literacy, 
as well as other important aspects of adult knowledge development, 
such as acculturation for immigrants and technical knowledge for 
vocational and professional education. (Contains 43 references.) 
(Author/KC) 
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Abstract 

Assessing adult literacy by telephone 



Recently, several industrialized nations have engaged in extensive and expensive 
assessments of adult literacy using door-to-door surveys and tests of "functional" reading. 
Following a review of research relating listening to reading, and a critique of the role of 
knowledge in literacy, this research examined the use of simple checklists of author's 
names, magazine titles, famous people's names, and vocabulary words used earlier by 
West, Stanovich & Miller (1993) to sample adult's declarative knowledge by telephone. 
Analyses revealed strong relationships among checklist knowledge (r=.80) when assessed 
either by listening (telephone) or by reading (mail out surveys). A comparison of findings 
using the telephone methodology with that of the recent U. S. Department of Education's 
National Adult Literacy Survey (NALS) showed that all major relationships of literacy to 
education, age, gender, occupation, income, ethnicity, father's and mother's education that 
have been found in the NALS and other literacy surveys over the last 75 years were found 
with the telephone methodology. Given the much lower cost of telephone versus door-to- 
door surveys (less than one sixth the cost), it is argued that the assessment of adult literacy 
by telephone is a promising and cost-effective methodological innovation that should be 
further developed as a means of monitoring national progress toward the achievement of 
greater levels of adult literacy, as well as other important aspects of adult knowledge 
development, such as acculturation for immigrants and technical knowledge for vocational 
and professional education. 



Assessing Adult Literacy by Telephone 



The assessment of adult literacy skills in industrial nations has taken on a new significance in the last 
decade. The Organisation for Economic Co-opcraiion and Development (OECD) notes mat, in regard to 
concerns for economic competitiveness among member nations, "... one area that is receiving growing 
attention from educational policymakers and analysts in a number of OECD countries is the direct 
measurement of literacy levels in the labor force of industrialized countries" (Benton & Noyellc, 1992, p. 1 1). 

The OECD report goes on to note that, wMlc some OECD countries, including the United States and 
Canada, have carried out national, direct assessments of adult literacy skills involving door-to-door surveys and 
the testing of thousands of respondents, the costs of such direct assessments arc extensive. Therefore, some 
nations may be reluctant to undertake such assessments, or, if they do, then they may resist performing follow- 
on assessments to determine whether adult literacy levels are improving. . 

In the United States, the National Governor's Association selected the National Adult Literacy Survey 
(NALS) administered in 1992 to measure progress in achieving national education goal number 5 - that all 
adults will be literate by the year 2(X)0. However, in a Memorandum prepared for a meeting of the National 
Governor's Association's Resource Groups for Goals 3, 4, and 5, it was indicated that there would be no 
follow-up to the 1992 NALS (Prince, 1994). In the discussions of the meeting of Resource Group 5 (now 
called Group 6, which one author of this paper attended as a member), it was noted that the NALS had been 
very expensive to administer, and hence it was not scheduled for re-administration. 

In a later telephone conversation with Dr. Andrew Kolstad at the U. S. Department of Education, 
National Center for Education Statistics, it was learned that the NALS had cost some S10.85 million to 
administer as a national survey, and with additional data collection for 12 ci the states in the U. S., the total 
cost of the NALS exceeded $14 million (Kolstad, 1994, personal communication). 

In Canada, too, concerns for the cost of administering direct assessments of adult literacy led the 
Department of the Secretary of Slate in Ottawa to explore the use of proxy measures of adult functional literacy 
(Ncicc, A'Jsctt, & Rodney, 1992, pp. 69-87). Using as a criterion the data from a 1989 national assessment of 
adult literacy skills, with items and procedures similar to those in the U. S. NALS, tnc Canadian team 
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investigated the use of lower-cost proxy measures such as self-reported assessments of skill, frequency of 
reading of newspapers, magazines, or books, years of education completed, and other factors related to literacy 
achievement to predict C.c direct assessment data. The investigators in Canada concluded that using such proxy 
measures, they mighl be able lo construct simulation models accounting for as much as 90 percent of the direct 
testing results (p. 84). 
Assessing reading by listening 

The present study, like that of the Canadian Secretary of State's study, was motivated by the need to find 
a more cost-effective method for assessing the literacy skills of adults than those used in the NALS or the 
Canadian direct assessments of household samples. But unlike the Canadian study of proxies for literacy 
assessment, the present study explored an alternative approach consisting of the assessment of one component 
of literacy, the knowledge used in comprehending during reading, but assessed by telephone without the use of 
written documents. Instead, the assessment of the knowledge component of literacy was based on the well- 
established relationship of listening to reading (Sticht, Beck, Haukc, KIciman & James, 1974; Sticht & James, 
1984; Sinatra, 1990; Stanovich, 1991). Sticht & James (1984, p. 297 ) surveyed literature comparing 
comprehension of various materials cither by listening or by reading. Averaging across numerous studies at 
different grade levels and into adulthood, they found thai the average correlations amon^ listening and reading 
increased from .35 in the first grade to around .65 in the sixth grade and remained at that level into adulthood. 

The data reviewed by Sticht & James (1984) were based on a variety of studies at each grade level and in 
adulthood. In these studies, the comparisons of listening and reading were made using materials that were not 
always carefully designed to make the listening and reading passages comparable. Additionally, task demands, 
such as the amount of time for listening versus raiding and the types of questions (fact; inference) were not 
carefully controlled. 

To overcome these limitations, research by Sticht, Hookc & Caylor (1981) used a specially developed 
lest batlcry lo compare the listening and reading comprehension of adults. In that study, brief paragraphs were 
developed for assessing listening and reading comprehension. The paragraphs were equated for difficulty using 
readability formulas and by magnitude estimation to equate passages on the basis of adults' subjective 
judgments of the difficulty of the passages. The time for reading passages was limited to the time needed to 



present the passages for listening. Only factual recall questions were asked and item statistics were used to 
ensure that the questions used to measure comprehension were of comparable difficulty in the reading and 
listening tasks. Under these carefully controlled conditions, the data for over 2,000 young adults with reading 
skills from the 2nd to the 14th grade levels produced correlations of .75 for listening and reading paragraph 
comprehension ( Stichl, Hookc & Caylor, 1981, Table 8, p. 39). 

The foregoing indicates that there is a body of research to suggest that listening and reading tests tend to 
rank adults in the same order. That is, low, medium and high ability adult listeners are likely to be low, 
medium and high ability readers, respectively. Therefore, an assessment of adults' listening competence may 
provide a valid estimate of their reading competence. The following discussion elaborates on some of the 
mechanisms involved in listening and reading that produce positive correlations among these two types of 
language information processing. 
A simple model of a human cognitive system 

In their formulation of research hypotheses regarding relationships among listening and reading skills, 
Stichlct al. (1974) developed a simplified version of the Atkinson and Shiffrin (1971) human information 
processing system model. In the simple /nodcl, the cognitive system was considered to include essentially two 
major subsystems. One of these subsystems is the long term memory with a store of content called the 
knowledge base. The other is the working mcmor> which is a short term memory that operates on the 
knowledge base using various processes including such language-based processes as listening/speaking and 
reading /writing. 

In this simple model of the human cognitive system, both listening and reading draw upon the same 
knowledge base (lexicon; syntax; pragmatics of language use; other episodic, schematic, semantic, declarative 
and procedural knowledge) and use it for producing or comprehending language whether the language is 
leprcscntcd in an oral or written mode. Dcvclopmcntally, the ability to represent knowledge by oral language 
generally prccccds the ability to represent knowledge by the written language. But ac lilcrac> develops, then 
both oracy and literacy converge upon the same knowledge base. 

In a direct vest of this convergence hypothesis using college undergraduate students, Sinatra (1990) 
concluded "... that listening processes and reading processes converge at the word level" (p. 126). She went on 
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to s lggcst that, "A student whose reading skills in listening comprehension and reading comprehension arc 
comparable may be reading as well as can be expected, and may be able to improve his or her reading ability 
only by building a larger vocabulary or a larger knowledge base" (p. 127). 

By the above reasoning, adults who possess low knowledge bases with relatively small vocabularies for 
representing their knowledge, will possess only limited ability to comprehend language-based messages 
presented in cither tho oral or written mode, and they will tend to be the poorly literate members of society. 
Conversely, those with large amounts of knowledge and the words for representing that knowledge will tend 10 
be the highly literate members of society. The fact that both listening and reading share a common knowledge 
base forms the basis for the approach to assessing literacy by telephone described herein. But first it is useful 
to understand some of the differences between literacy and oracy that may tend to lower empirical associations 
among listening and reading assessments. 
Listening and literacy: some differences 

Though there is considerable comparability among listening and reading, the information displays used in 
speaking and writing arc not the si me and so relationships among these two types of receptive language 
processes arc not perfect. The information displays used in presenting spoken and written language differ 
greatly and permit the development of knowledge and information processing strategies and skills that differ for 
listening and reading. Sticht (1978) and Sticht & McDonald (1992) have distinguished between literacy 
considered as a second signaling system for speech, and literacy as a special case of the use of graphics 
technology to aid communication, reasoning, and problem solving. 

The positive relationships among listening and reading tests reflect the use of graphics technology to 
produce a graphic display called "written language" that can serve as a second signaling system for speech. 
What can be spoken can be written, whai can be wriltcn can be read, what can be read alone! (i.e., spoken) can 
be comprehended by listening. 

The second aspect of literacy recognizes that graphic displays possess elements that permit people to 
develop knowledge and skills beyond that used in oral language. The graphic information display is more or 
less permanent, it can be arrayed in space, and it uses the properties of light (contrast; color). These properties 
arc drawn on by literates to design information processing tools such as more or less permanent pages of print 
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(knowledge bases slorcd outside ihc head) with bold (use of light: contrast) headings to direct attention to 
important concepts, with information arrayed spatially to assist comprehension of complex relationships, as 
in a train schedule with numerous intersecting rows and columns, or other information presentation devices 
(tables, figures, charts, schematics, etc. ). 

Because of these properties of graphics displays, literates n ay acquire knowledge about such displays and 
information processing skills for using such displays that go beyond the use of written language as a second 
signaling system for speech. The relative permanence of graphic displays permits tnc development of various 
information "search and locate" processes (Guthrie, Britten, & Barker, 1991) that arc not possible with the 
temporally fleeting spoken language (though audio recording devices have made possible the use of 
information "search and locate" strategics for recorded speech, but these processes arc not relevant to the present 
discussion). 

The National Adult Literacy Survey (NALS) in the United States used literacy tasks that were based, to a 
large extent, on a theory of "document literacy" that emphasi/.cd "search and locate" information processing 
skills (Kirsch, Jungcblut, Jenkins, & Kolstad, 1993, p. 9; Moscnthal & Kirsch, 1991, pp. 147-179). The 
NALS report states that, "On the prose scale, for example, tasks with low scale values ask readers to locate or 
identify information in brief, familiar, or uncomplicated materials, while those at the high end ask them to 
perform more demanding activities using materials that tend to be lengthy" (italics added) (Kirsch, et al M 19vi, 
p. 9). 

In terms of the simple model of a human cognitive system, locating information in texts or complex 
documents places demands on working memory because information that is to be searched for has to be held in 
working memory while simultaneously bringing into working memory information from the document to 
determine whether it matches the sought information. In general, increases in the amount of information that is 
being sought and the amount of information that must be searched to locate the correct information place 
greater demands on working memory and greatly increases the difficulty of tasks (Meyer, Marsiskc, & Willis, 
1993, pp. 234-249; Kirsch & Moscnthal, 1990, pp. 5-30). 

Knowledge, information processing and literacy 

Literacy assessments such as the NALS engage the respondents in tasks that involve much more than 
comprehending written language as a second signaling system for speech, though they also include this aspect 
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of literacy, too. The use of tasks thin tend to overload working memory while simultaneously engaging the 
reader in comprehending various domains of knowledge creates a scale of literacy tasks of increasing difficulty 
that systematically differentiates readers into low, medium and higher skilled literates. According to the theory 
that underlies the creation of the NALS items and scales, literacy tasks grow more difficult as (1) there is more 
information in the document, (2) the tasks require more categories to be processed, and (3) the tasks require 
significant use of problem-solving ability or unique prior knowledge to relate different types of information in 
the tasks (Mosenthal & Kirsch, 1991, p. 175). 

In terms of the simple model of a human cognitive system given above, what this means is that the 
performance of increasingly difficult tasks such as those on the NALS depends both upon the possession of a 
broad knowledge base to understand task directions and to comprehend the various contents of the materials, and 
efficient information processing skills that operate in working memory to hold instructions, perform search and 
locate actions, and problem solve (make inferences; reason) to accomplish tasks. 

A five-year research program known as the Learning Abilities Measurement Program (LAMP), conducted 
by researchers at the Air Force Human Resources Laboratory (AFHRL-now called the Armstrong Laboratory) 
explored the cognitive skills that comprise reading comprehension and other general learning abilities (Payne, 
Christal, & Kyllonen, 1986). A major outcome of the first five years of the project was the finding that 
individual differences in reading and reasoning were predictable by the capacity of working memory, the breadth 
of the declarative and procedural knowledge base, and the speed with which these cognitive subsystems and a 
perceptual subsystem operate (Kyllonen & Christal, 1990). Generally speaking, highly literate individuals 
possess large bodies of knowledge and efficient information processing in working memory to process 
information in complex graphic documents. 

However, though large bodies of knowledge and efficient information processing skills typically go 
together, the graphic display being somewhat permanent, makes it possible for some with less efficient 
information processing skills (e.g., slow word recognition) to nonetheless perform fairly complex literacy 
tasks because the display persists and can be repeatedly searched and studied to locate and extract information. 
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Of course, adulLs who arc practically completely illiterate may perform by listening tasks that they 
simply cannot do by reading. For these groups, the correlation of listening with reading will be very low, if 
not zero, because there will be little variation in reading es. However, as a group, these same persons will 
tend to perform at the lower end of the distribution of listening when a full range of educated and literate adult 
listeners arc assessed. Hence, their relatively low listening scores will predict their low literacy skills, too. 

Additionally, because people who spend a lot of time studying and learning broad bases of knowledge 
typically do much more of this using texts than by listening, they may recognize some words when they sec 
them in writing but not when they hear them spoken. These differences in the auditory and graphics displays 
and the differences in information processing they afford may serve to reduce correlations among listening and 
reading tasks. 

What makes people highly literate? 

As noted above, generally speaking, highly literate individuals possess large bodies of knowledge and 
efficient working memory processes to decode written language. But how do the highly literate obtain these 
vast bodies of knowledge and efficient information processing skills, such as those used in the automatic 
recognition of written words? 

Several lines of research have converged to suggest that people become highly literate largely by 
engaging in numerous literacy practices (Rcder, 1994, pp. 33-74), such as reading books, magazines, 
newspapers, and so forth. A review of the major assessments of adult literacy in the United States revealed that, 
since 1937 it has repeatedly been found that for adulLs, as years of education increases there arc corresponding 
increases in both the number of literacy practices in which adulLs engage and the amount of skill displayed in 
the assessments (Slicht & Armstrong, 1994, pp. 43,63,99,1 13). 

In an illuminating program of research, Keith Stanovich and associates have explored how the extent of 
engagement in literacy practices by children and adults has contributed to their development of literacy and, 
more broadly, "verbal intelligence" (sec Stanovich, 1993 for a general summary of much of this research; 
Stanovich & Cunningham, 1993). The present study uses the methods developed by Stanovich and associates, 
and therefore their research program is discussed in some detail. 
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Using an innovative method for assessing knowledge with checklists that call for a simple "yes" or "no" 
judgment on the part of the reader, Stanovich and associates have demonstrated that performance on these 
checklists correlates significantly with a variety of literacy activities and assessments. 
Reading out of school: children 

Amount of out-of-school reading time that fifth grade children reported in activity diaries was 
significantly correlated with scores on various checklists asking for knowledge about book authors, book titles, 
and other kinds of knowledge. The correlation of reported amount of time spent reading books during the day 
with scores on an Author Recognition Test (ART) consisting of a checklist of names, some of whom were 
authors and some of whom were not, was .5?.. Corrected for attenuation due to low reliabilities of diary reports 
and the checklists, the correlation was .70. The uncorrected correlations of two forms of a Title Recognition 
Test (TRT) with the diary reports of lime spent reading were .48 and .43, and these correlations rose to .65 and 
.59 when corrected for allcntialion. (Allen, Cipiclcwski, & Stanovich, 1992, pp.496-497). A major conclusion 
from this research was that the simple checklists could serve as easy to obtain, low-cost proxies for the more 
expensive and difficult to obtain diary studies of the amount of print to which readers expose themselves. For 
this reason, Stanovich and associates refer to the various checklists as measures of print exposure. 
Reading out ofsclwol: aduhs 

In a study of adult "real world H reading, West. Stanovich, & Mitchell (1993, pp. 35-50) observed 
passengers wailing for their flights in National Airport, Washington, DC, USA. If the subjects engaged in 
reading for 10 minutes, they were designated as "readers" and if they did not read for 10 minutes they were 
designated "nonrcadcrs." Both groups were approached to determine if they would participate in a study in 
which they completed a set of simple checklists including an Author Recognition Test (ART), a Magazine 
Recognition Test (MRT), a Cultural Literacy Test (CLT) consisting of names of people, some of whom were 
famous and some were not, a Vocabulary Recognition Test (VRT) consisting of a list of real words and some 
pronounceable, non-words, and several other checklists calling for knowledge of tclcvison programs, telcvison 
newspersons, and so forth. The major finding was that there were significant differences between the "readers" 
and "nonreadcrs" in their knowledge as assessed by the various checklists, with the "readers" out-performing the 
"nonrcadcrs." This confirms the findings with the fifth grade children that the checklists are useful for 
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distinguishing those who expose themselves to print by engaging in greater amounts of literacy practice (the 
"readers") from those who engage in less amounts of such practice (the "nonrcadcrs"). 
Cognitive correlates: children 

For the fifth grade students discussed above, significant, positive coefficients were found in 20 out of 24 
correlations of print exposure checklists with various measures of verbal ability, including vocabulary, general 
knowledge, Wechslcr Intelligence Sca 1 ^ for Children-Revised, and reading comprehension. Significant 
correlations ranged from a low of .28 to a high of .63 between different checklists and different criterion 
indictors of literacy (Allen, Cipiclcwski, & Stanovich, 1992, p. 498, Table 5). 
Cognitive correlates: adults 

Studies with college students also revealed a positive manifold of correlations among print exposure 
checklists including an Author Recognition Test (ART) , a Magazine Recognition Test (MRT) , a Newspaper 
Recognition Test (NRT), and a composite score for these checklists, with a wide variety of general ability 
measures and measures of knowledge (Stanovich & Cunningham, 1993, Table 1, p. 216). The general ability 
measures included high school Grade Point Average, the Nelson-Denny Reading Test, the Raven Advanced 
Progressive Matrices and a mathematics test. Significant correlations ranged from .20 to Al with the 
checklists and these various measures of general ability. 

The measures of knowledge included a multiple choice test of "cultural literacy," with items assessing 
knowledge in science and other topics, a practical knowledge test (If a substance is carcinogenic it means that 

it ?), a cultural knowledge checklist, which included names of famous people from movies, history, 

a? \ etc., a multicultural knowledge checklist to overcome male and European bias in cultural knowledge, an 
acronym test and a composite of these tests. Importantly, the ART, MRT, NRT and print exposure composite 
checklists correlated even higher with these knowledge tests than with the general ability tests. Significant 
correlations ranged from .53 to .81. The correlation of the print exposure composite scores with the general 
knowledge composite scores was .85. Interestingly, the correlation of the cultural with the multicutural 
checklist was .78. This, plus the fact that the intcrcorrclations among the various measures of knowledge were 
themselves all positive, indicates that highly literate individuals tend to possess great breadth of declarative 
knowledge, as suggested in the LAMP project reported above. 
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In the airport study, analyses revealed that among both the "readers" and "nonrcadcrs" scores on the print 
exposure measures af author, -nagazinc title, and newspaper title recognition contributed significantly above 
and beyond age and education variables to the prediction of scores on vocabulary and cultural literacy checklists, 
when these were treated as outcome or dependent variables (West, Stanovich, & Mitchell, 1993, Table 6, p. 44 
& Table 7, p. 45). 

From a causal perspective, across these varu.us studies, the argument by Stanovich and associates is that 
those who read a lot acquire, mostly through incidental learning, a large declarative knowledge base containing 
the names of authors, magazines, newspapers, persons known for their contributions to film, theatre, music 
and other cultural activities, and a large vocabulary of words that arc typically not encountered with high 
frequency in day-to-day oral communication nor on television or radio. Scores on these various checklists arc 
indicators of both the amount of reading in which individuals engage and of the cognitive outcomes of that 
reading in terms of the growth in the individual's declarative knowledge base. 

Age and education arc related to performance on these checklists for at least two reasons. First, will, more 
age people have had more lime to complete more years of education and to engage in incidental learning by 
reading. Second, education directs people into various domains of reading and this helps develop the 
vocabulary and other declarative knowledge needed to read more broadly and with greater comprehension and 
interest. 

Using the Stanovich checklists to assess literacy by listening 

The checklist approach to knowledge assessment developed by Stanovich and associates lends itself 
readily to the assessment of knowledge by listening. Based on the ideas of reading as a second signaling system 
for listening and the convergence hypothesis, that is, that spoken and written language converge at the word 
level of proccsssing (Sinatra, 1990), it seems likch dial one could assess the knowledge base that a person 
draws upon in performing reading Lasks by asking the person about his or her knowledge of literacy sources 
(authors; magazine titles) and information that may be gained by wide-ranging reading (famous persons or 
events from history, the ails, and science; vocabulary words). 

The use of simple items such as names or single vocabulary words, with each yes/no decision made 
independently of the other does not overload working mcmoiy. This is an especially important factor when 
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assessing the literacy of elderly adulls. In tt U. S. Naiionai Adult Literacy Survey (NALS), data for 
f)erformancc ur< prose, document, or quantitative scales indicated thai, depending on which scale is discussed, 
performance gradually improved by about .16 10 .36 standard devialions as age increased from 16-18 through 
40-54 years. However, above age 54, there was a ^apid decline of about a half standard deviation for those 55-64 
years, and over one standard deviation for those 65 years and older (Kirsch, Jungcblut, Jenkins, & Kolstad, 
1993, p. 31). Since it is well established that working memory becomes increasingly less efficient with 
advanced age (Bernstein, Roy, Srull, & Wickcns, 1988, p. 401; Meyer, Marsiskc, & Willis, 1993, p. 235), 
these findings strongly suggest that the NALS tasks derive a great deal. of their difficulty from the load they 
place upon working memory. Hence they may seriously underestimate the breadth of materials that oldei adulls 
can read and comprehend using ihcir knowledge base and the tasks they can perform in working memory given 
sufficient time to study materials and without the pressure for efficiency that is typical of test-taking 
situations. 

The assessment of knowledge as distinct from complex literacy task performance is also warranted when 
the definition of literacy that was adopted by the advisory panel of experts for the NALS is examined. The 
definition of literacy agreed to was as follows ((Kirsch, Jungcblut, Jenkins, & Kolstad, 1993, pp. 2-3). 



The importance of knowledge in literacy was also acknowledged by the advisory panel for the NALS in 
its acceptance of the definitions of the three different literacy scales that were developed (p. 3, italics added in 



information from texts...." 

"Document literacy - the knowledge and skills required to locale and use 
information contained in materials...." 

"Quantitative literacy - the knowledge and skills required to apply arithmetic 
operations.. .embedded in printed materials." 



From these definitions, ii is clear that the advisory panel for the NALS understood that the use of printed 
and written information to accomplish tasks requires, as a prerequisite, certain knowledge and skills to make 



"Using printed and written information to function in society, to achieve one's 



goals, and to develop one's knowledge and potential (italics added)." 



each case): 



Prose literacy -the knowledge and skills needed to understand and use 
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such use possible. Strictly speaking, therefore, literacy is not "using printed or written material to function in 
society, etc" as given in the advisory panel's agreed upon general definition. Rather, literacy is the prerequisite 
knowledge and skills thai make the use of printed and written materials possible. 

As indicated above, the Stanovich checklists make possible f.hc assessment of knowledge that is at one 
and the same lime indicative of (1) the extent of use that one makes of printed and written materials, (2) the 
knowledge that one h> s derived from the use of these materials, and (3) the added potential for engaging in 
future literacy practices as a consequence of having a broader base of knowledge prerequisite for wide-ranging 
reading. 

The simple checklists arc indicators of the extern to which adults have used written or printed information 
in the past. By examining relations of the chcck ,; sl scores to other information, such as number of years of 
education completed, occupational standing, and income, one should be able to estimate the degree iO which 
printed materials may have been used as in the NALS definition, i.e., to function in society to achieve one's 
goals in education, work and economic standing. 

The following study investigated the assessment of the knowledge component of adult literacy using 
telephone interviews as the mode of presentation, listening as the mode of reception, and the Stanovich 
checklists to assess aspects of the declarative knowledge base that arc likely to be developed by reading. To 
evaluate the hypothesis that reading and listening converge on the same internal knowledge base, including 
language as a part of the knowledge base, a mail on: ~» survey was sent to a subset of the telephone 
survey sample and the correlations of scores obtained by reading the written survey and by listening to the 
telephone survey were computed. 

To permit comparisons of the results of the telephone survey to the findings from other surveys of adult 
literacy, especially the NALS, the interview included questions to obtain data on the relationships of literacy 
knowledge (i.e., scores on the checklists) to factors that have been shown in the past to relate to adult literacy, 
including education, age, primary language, ethnicity, engagement in literacy practices (c.g, reading books, 
magazines, newpapers, etc.), intergencralional relationships of fathers and mothers years of education to the 
respondent's literacy knowledge scores, occupational status, and income. 
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Method 

Subjects 

Data for this study were derived from telephone interviews with 538 adults residing in households that 
could be reached by listed or unlisted telephone in the larger San Diego, California metropolitan area.Thic 
included approximately 96 percent of all households. Sampling was conducted by using a Random-Digit- 
Dialing procedure designed to reach households without numbers listed in the telephone directory, due to 
unlisted numbers or newly listed numbers not yet printed, as well as households listed (Dillman, 1978, pp. 
232-281; Frcy, 1989, pp. 79- 1 16). A sub-sample of those reached by telephone who agreed to participate was 
included in a mail-out, follow-up survey. 

In the telephone interviews subjects reported averages of having lived in the San Diego area for 20.6 years 
(SD=15.5 years), having completed 14.5 (SD=2.6) years of education, being 41.0 (SD=16.0) years of age, 
earning a total household income ol'S34, 340 (SD=S12,240), and having an average of 3.0 persons (SD=1.8) 
in the household. 

Similarity of the survey sample to U. S. Census data 

The survey procedures resulted in a sample that matches 1990 U. S. Census dam closely, with several 
notable exceptions. Table 1 show's statistics for the telephone and mail-out surveys and U. S. Census 

I insert table 1 about here] 

population parameters for the San Diego region. Telephone and mail-out samples of marital status, gender, 
age, and income were similar 10 census distributions. The telephone and mail-out sample data were skewed 
upward in educational attr'nmcnt. The lowest level of of educational attainment is underrcprcscntcd and the 
highest level is ovcrrcprcscnlcd among the sample data in comparison to the census data. Because the less 
educated arc likely to have lower scores on literacy assessments, the low numbers of less educated adults in the 
telephone and mail-out samples may bias population literacy estimates upward somewhat. Regarding ethnic 
distributions, whites arc slightly ovcrrcprcscntcd and minorities slightly underrcprcscntcd in the telephone and 
mail-out samples in comparison to the census data. There arc, nonetheless, sufficient numbers of minorities to 
permit an estimate o. literacy levels by ethnicity for this demonstration study. 
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Procedures 

Telephone interviews 

Interviewing was conducted by university students who had been trained for telephone interviewing for 
the project during the late spring and early summer, 1 994. Subjects were called between 4:30 p.m. and 9:30 
p.m. weekdays and between 9:00 a.m. and 9:30 p.m. on weekends. Interviewers introduced *Jie survey to the 
person who answered the telephone, gained informed consent, and asked to speak to the adult (18 years of age 
or older) who had "the most recent birthday" as a method of selection among adults in the household. No 
substitutions were allowed so that interviewers frequently were required to call the household back in order to 
complete an interview with the appropriate respondent. Up to four callbacks were made to residential 
households and a response rate of approximately 50 percent was attained. Due to resource constraints, 
interviews were conducted only in English, a procedure that eliminated approximately 4 percent of households. 

The telephone interviews provided an oral presentation of information which required the respondents to 
listen and respond from what they heard. Interviewers followed a protocol containing 63 questions, some with 
multiple sub-questions. About half of the questions were concerned with the assessment of literacy. The 
literacy-related questions were interspersed amoni' other questions that were asked as part of another on-going 
research project conducted in the area of political science. Those questions and their responses arc not included 
in this study. The average interv iews required 27.7 (SD= 7.6) minutes to complete. 
Mail-out survey 

At the end of each of the telephone interviews, subjects were told: u My office may wish to contact people 
who have helped us in this project to ask a few more questions. by mail. May I include you on this list?" 
Subjects who responded affirmatively to the request (n=280) were then mailed a qucstionairc consisting of paper 
and pencil versions of the four literacy assessment checklists, a 26 item cloze test developed by Mikulccky & 
Dichl (1980, p. 78) thai was used to assess the general literacy skills of employees in 100 occupations, with a 
scale for converting doze raw scores to reading grade levels, and several items calling for demographic 
information. 

Questionaires were sent under the university letterhead of one of the authors with a letter reintroducing the 
project and requesting that each subject "take a few minutes" to fill out the qucstionairc and return it to the 
research office. Two waves of mailouts were used, the first occuring 7-14 days following the initial telephone 
interview, and a follow up mailing to those who had not responded within 21-28 days. About 50 percent 
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(n=140) returned usable qucstionaires. From several small-scale pilot tests with subjects ranging from those 
enrolled in an adult basic education ^!ass to college graduates, it was estimated that 10-15 minutes would be 
needed to complete the written survey. 
Instrumentation 

As mentioned above, the telephone survey involved a number of questions that were not relevant to this 
study. Thos^ that were relevant were demographic questions, including years of education, age, ethnicity, 
gender, English and other language use, occupation and income. The appendix presents a list of additional 
questions that provide information similar to various categories of information obtained in the National Adult 
Literacy Survey (NALS). 

Tlic literacy knowledge checklists 

For the sake of time, four abbreviated versions of the checklists used by West, Stanovich and Mitchell 
(WSM) (1993) were used in ihc telephone survey. The appendix shows the items used in the Author 
Recognition Test (ART) (Q26), the Magazine Recognition Test (MRT) (Q27), the Cultural Literacy Test 
(CST) (Q3 1) and the Vocabulary Recognition Test (VRT) (Q36). 

For the ART, 10 actual author's names and 5 foil names taken from the 25 items in the WSM study were 
used. The items were chosen to give a good range of difficulties. Similarly, the MRT was comprised of 9 
actual and 6 foils from the 25 items used by WSM. The CLT was made-up of 17 actual and 6 foils from the 
WSM list of 45 items, and 14 actual and 7 foils were taken from the list of 40 vocabulary items used by 
WSM. The foils for the VRT were pronounceable non words. 

For each checklist, the score was the proportion of correctly identified real names or words minus the 
proportion of foils identified as bc'.ng real names or words. For instance, if a person said "yes" to 10 of the 17 
names of famous people on the CLT and to 2 of the 6 foils, the person's score for the CLT was (10/1 7) minus 
(2/6) or 58.8 minus 33.3 equals 25.5. The reason for the correction lor guessing was to prevent people from 
simply saying "yes" to all items. The rationale and references regarding the scoring procedures arc given in 
WSM (p. 38). The appendix shows the scores corrected for guessing for each item on the four checklists, 
along with corresponding data from WSM for those designated as "readers" and "nonrcadcrs" (there wc:c no 
scores provided for the vocabulary checklist in WSM). 
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Split-half, internal consistency (Spearman-Brown ) reliabilities of the checklists ranged from .80 (MRT) 
to .88 (CLT). To increase the reliability of the checklists as measures of the knowledge component of literacy, 
a Total score was calculated made up of the fall number of 50 actual and 24 foil names and words. The internal 
consistency reliability for Total literacy was .91 . Table 2 presents tcst-rctcst, "alternate modalities," stability 
reliabilities for the Total score (.80) and each of the checklists, ART (.71), MRT (.67), CLT (.73), and VLT 
(.63), obtained 'indcr the telephone (listening) and mail-out (reading) conditions for 140 respondents. Thus, 
strong evidence of reliability, both in terms of internal consistency and test-retest, was present for each of the 
four scales and the total scale. 

Engagement in literacy practices! print exposure 

Questions were asked regarding the number of times in an average week the subject engaged in various 
literacy practices such as reading for pleasure newspapers, hooks, and newsmaga/Jncs or reading job-related 
materials for work (Q7;Q25). Question 25 also asked fo: frequencies per week that the person engaged in 
reading to a child or listening to someone read aloud. This is related to the interest in the intcrgcncrational 
transfer of literacy, but in this case the interest was in the transfer of literacy from the respondents to their 
children. 

Self report indicators of literacy competence 

Two questions asked the respondents to rate their skills in reading in general (Q28) and reading to meet 
the requirements of their jobs (Q40). Self perceptions of skills were also obtained in questions that asked 
subjects whether they thought they could get a better job if they received additional training in reading and 
writing English (Q45) or in mathematics (Q46). Ratings of how respondent's competence in understanding 
what they read at work were obtained (Q30a,b,c,d,c). Additionally, indirect self-reported indicators of literacy 
skills were obtained in a scries of questions about the extent to which the respondent received help from family 
or friends in reading and/or writing various materials (Q33a,b,c,d,c). 

Inter generational relationships 

Two sets of questions provided information about subjects' childhood home influences on their present 
literacy. Questions 34 and 35 each asked for information about the subjects' literacy environments when they 
were in high school (e.g., Q34c\ When in high school did you have more than 25 books in the home?) and 
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whcihcr they had ihc same lite? \cy resources in iheir homes al the lime of the interviews. Information 
regarding the respondents' fathers' (Q53) and mothers' (Q54) years of education permitted the determination of 
the intcrgcncraiional relationships among parents 1 education levels and the subsequent education and literacy 
achievement of their adult children. 

Results 

The results of the mail-out survey in comparison to the telephone survey arc presented first, to confirm 
l!ie relationships among listening (telephone) and reading (mail-out) discussed above. 
Listening and reading 

Some 140 of the 538 adults reached in the telephone survey completed and returned a mail-out survey that 
contained the same knowledge checklists as used in the telephone survey. Analyses revealed that the mail-out 
subsamplc was slightly belter educated than the lull telephone sample (mean years of educalion=15.01, S.D.= 
2.39 versus mcan=14.53. S.D. = 2.62 lor the total telephone sample). Respondents to the written version 
scored somewhat higher (mean percent correct for Total literacy = 61 , S.D. = 23) ti in they did on the telephone 
survey (mean percent correct for Total literacy - 57; S.D. = 22) 

Table 2 presents the correlations among the literacy knowledge checklists when completed by the same 
140 subjects by listening in the telephone interview and some two to three weeks later by reading the mail-out 
survey. The underlined coefficients arc the alternate modality, icst-rcicst reliability scores resulting from 
correlating checklist scores obtained by listening to the telephone interview with the scores on the same items 
obtained by reading the mail-out survey. These correlations arc in the range of those reviewed earlier 
establishing relationships among listening and reading. They indicate that adults who scored low, medium, or 
high on the checklists when listening in the telephone interview tended to maintain their relative rank orders 
two to three weeks later when they completed the checklists by reading on the mail-out survey. 

(insert table 2 about here] 

Validil\ cod I 'ic tonus are given in Table 2 as correlations of the checklist scores obtained by listening and 
by reading with the scores on the cloze lest that was included in the mail-out survey. All the correlations 
among checklist and doze scores were positive and statistically significant. The predictive validity coefficients 
for checklist knowledge and clo/e scores, resulting from correlating checklist scores obtained by listening two 
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to three weeks before completing the cloze test on the mail-out survey, were only slightly lower than the 



concurrent validity coefficients obtained with the cloze tests and the written checklists when completed 
together on the mail-out survey. 
The telephone survey results 

Table 3 presents the correlations among key demographic variables, the checklists and a "practice" 
variable (e.g., How often during an average week do you read a local or national newspaper?) computed as the 
average of questions 25a-k (see appendix) . The practice variable is an indicator of "print exposure" and relates 
average frequency of weekly reading of different materials for various purposes to education, age, and the 
knowledge checklists. [insert table 3 about here] 

In Table 3, the data from the telephone survey arc presented above the diagonal, while for comparison 
purposes below the diagonal, data from West, Stanovich, & Mitchell (WSM) (1993) arc presented . Overall, 
there is remarkable consistency between the findings of the telephone survey and the work of WSM. In both 
studies, education is positively related to the amount of knowledge of authors, magazines, famous people and 
vocabulary indicated by scores on the checklists. 

Table 4 presents the means and standard deviations of Total literacy scores for six major demographic 
variables. The data of Tables 3 and 4 show positive correlations among demographic and literacy variables that 
have been consistently found in adult iitcracy assessments for over 75 years (Sticht & Armstrong, 1994). 
Better educated respondents scored higher than the less well educated, respondents with better educated parents 
scored better than those with less well educated parents, the majority group (whites) scored better than 
minorities (Hispanics, Blacks, Asians, others), managers and professionals performed better than clerical and 
sales persons, who, in turn, performed better than unskilled workers and laborers, those who earned more scored 
higher than those v\ ho earned less, and those who spent more time per week reading scored higher than those 
who read less. [insert table 4 about here] 

Engagement in literacy practices! print exposure 

The appendix presents mean scores and SD's for questions 7 and 25a-k dealing with respondent's estimates 
of the frequency with which they engaged in various literacy practices in a typical week. Overall, respondents 
reported that they read a newspaper 4.4 times a week (SD=2.8) (Q's 7 & 25g). Reading for pleasure (Q 25a) 
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was the most frequent reading practice (M=4.68; SD=2.50) while listening to someone read aloud was the least 
frequently engaged in weekly literacy practice (M=0.f 2; SD=0.74). 

To obtain a summary indicator of the frequency of weekly engagement in literacy practices, the responses 
to questions 25a through k were summed and averaged. Tabic 5 presents these averages and SD's for various 
demographic groups. Generally, the trends for practice follow those for Total literacy (Table 4). As education 
(r=34) , age (r=.08), occupational status (r=.17), income (r=.26), and father's education (r= .08) increase, the 
average frequency of weekly literacy practices increases (though the correlation of father's education and literacy 
practice were not significant at rx.09). Whites were slightly more likely to engage in literacy practices than 
nonwhitcs (r=.()9). 

(insert tabic 5 about here] 

Questions 5A and 6 in the appendix present the average hours per day that were reported to be spent either 
watching television or listening to the radio. There was a significant, negative (-.14, rx.OOI) relationship 
among the number of hours of television watched and the average weekly literacy practice score. No 
relationship of radio listening to literacy practices was found. Neither television viewing nor radio listening 
were related to any of the literacy knowledge checklist scores. 

Factor analysis of praclic c items 

To belter understand relationships of literacy practices to demographic and literacy knowledge variables, 
the items in question 25a through k were subjected to a principal components factor analysis and loadings were 
rotated to simple structure by varimax criteria. Tabic 6 lists the factor loadings greater than .30. The four 
factors extracted account for over 62% of the total variance. 

[insert table 6 about here] 

Practice J (News) groups the questions dealing with newspaper and magazine reading. Practice 2 (Job) 
groups the reading at work items. Practice 3 (Pleasure) groups the items pertaining to the reading of books and 
other materials for pleasure, while Practice 4 (Family) groups items that suggest literacy practices for parenting 
or social communication around the home. 

The correlations of Table 7 provide information to aid in the interpretation of relationships among 
literacy practices and various demographic variables. For instance, literacy Practice 4 (Family) is significantly 
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and negatively related lo age, suggesting that elderly adults do not read to or listen to children read very much; 
positively to gender, suggesting that females arc more likely to read to others than arc males; negatively to 
Latino status, suggesting that Latinos arc less likely to engage in the types of reading in practice 4; and 
negatively to Total literacy, the Author Recognition Test, and the Vocabulary Literacy Test (VLT), suggesting 
that reading to or listening to children or others read aloud docs not contribute much to the growth of 
knowledge or vocabulary as measured in the checklists. In all cases, however, these Practice 4 (Family) 
correlations are very low and do not appear to account for much variance in the variables examined, 
[insert tabic 7 about here] 

Interestingly, literacy Practice 3 (Pleasure), reading books or other things for pleasure is consistent)' the 
highest correlate with the various literacy knowledge checklists. Literacy Practice 2 (Job) , reading on the job, 
is the only factor significantly related to both father's and mother's education. Literacy Practices 1 (News) and 3 
(Pleasure) , reading newspapers, magazines and books for pleasure arc positively related to age, while reading 
for work and for parenting arc negatively related to age. This suggests that older people read more for pleasure 
and information, while younger people tend to read more for work and parenting. Women were less likely to 
read as much as men in the newspaper sections asked about in literacy Practice 1 (News) or the types of 
materials asked about on the job (Practice 2). 

Self -report indicates of literacy competence 

Questions concerning self-reports of literacy competence showed a positive manifold of significant 
correlations with literacy checklist scores in the expected directions. For instance, as Total literacy scores 
increased, respondents were more likely lo say that they understood nearly all or all of what they read (Question 
28 in the appendix, r=.35, p< .001 h that their reading ;kills were adequate or more than adequate for their job 
(Q40, r=.34, p< .001 ), that reading or writing training in English would not help them get a better job (Q45, 
r=.26, p< .001), and that additional training in mathematics would not help them ^el a better job (Q46, r=.21, 
p< .001). Similar, though somewhat lower significant correlations were found with the four separate 
checklists. Those who engaged in more literacy practices (average of Q's 25a-k) rated themselves able lo 
understand more of what they read (Q28, r=.27, p< .001 ). 
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The frequency of responses 10 the raiing scales for question 28 indicated that 6.7 percent of respondents 
thought they sometimes had trouble understanding what they read, but four percent thought that their reading 
skills were cither "not at all adequate" or "not as good as they should be" to do their job (Q 40). 

Consistent with these positive self- perceptions of reading skills, question 33 asked respondents to 
estimate how often they had help from family members or friends in accomplishing a variety of literacy 
practices and most reported that they never had any help. However, when it came to dealing with government, 
business, medical , etc. agencies or personnel, about 15 percent reported that they got help on a monthly basis 
from friends or relatives. Less than 2 percent reported receiving help on one or another literacy practice on a 
daily basis. 

Despite these generally optimistic ratings of reading ability, there were some indicators that many 
respondents may have had a low sense of security about their reading abilities. Over 10 percent disagreed or 
strongly disagreed that they could read well enough to do their jobs well even when there were distractions (Q 
30d), and almost one in ten (9.4 percent) disagreed or disagreed strongly that they could read well enough to do 
their job when under pressure to meet a dead I i ic (Q 30e). 

Many seemed to place a high value on being able to read better than they do. One in six (14.4 percent) 
thought they would get more respect at work if they understood what they read belter (Q30a). One in five (19.4 
percent) thought they would be able to do a better job if they could read better (Q30b), and one in eight (13.1 
percent) thought thes could earn more money i! they could read better (Q30c). 

Inicrgcncraiional relationships 

Father's education was positively and significantly related to Total literacy (r=.l(), p< .03), MRT (r=.14, 
p< .002), VLT (r=.l 1 , p< .014), and literacy practice 3 (reading on the job) (r=.20, p< .001). Table 4 shows 
the means and SD's on Total literacy by levels of father's education, while table 5 shows the means and SD's 
for father's education levels and total practice (average of Q's 25a-k) score. Mothers education was positively 
and significantly related only to MRT (r=.18, p< .002) and literacy practice 3 (reading on the job) (r=.13, p< 
.002). 
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Establishing Literacy Levels 

Both the Armed Services and the National Adult Literacy Survey have cast distributions of scores on 
literacy assessments into five levels of proficiency to identify groups of the least to the most proficient 
literates (Slicht & Armstrong, 1994). Similarly, then, to illustrate the feasibility of that widely used approach 
in the present case, the results of the telephone survey were cast into five levels of proficiency using Total 
Literacy scores. 

The five levels were obtained using the mean (45) and standard deviation (25) of the percent correct scores 
for the Total Literacy distribution. Literacy levels were defined from low to high proficiency as: Level 1 = 
scores at -1.0 SD or lower (0-20), Level 2= scores between -.5 to -1.0 SD (21-32), Level 3= scores between ± 
.5 SD (33-58), Level 4= scores between +.5 to +1 .0 SD (59-70), and Level 5= scores from +1.0 SD and above 
(71-100). 

Table 8 gives the percentage of the telephone sample that fell into each of the five levels for the full 
sample as well as for various groups within several demographic variables. For comparison purposes data arc 
presented showing the perccnuigc of cases under the portions of the normal curve that were used to define the 
five literacy levels, the percentage of cases falling within each of the Armed Service's five categories for the 
Armed Forces Qualification Test (AFQT) based on the 1980 rcnorming of the Armed Services Vocational 
Aptitude Battery (Eitclbcrg, 1988, p. 101), and the percentage of cases falling within each of the five literacy 
levels defined by the National Adult Literacy Survey (NALS) (Kirsch, ct. al, 1993). It should be noted that the 
Armed Services label their five levels from 5 as the lowest to 1 as lhc highest proficiency. Here we have 
reversed the numbers to be consistent with the present survey and the NALS. Also, to represent the NALS, we 
used the median percentage for each level for the three scales of prose, document and quantitative literacy. 

(insert Table 8 about here) 

Given Luc vast differences in materials, tasks, contexts, samples, and procedures for scoring and grouping 
people, there is little reason to expect any great similarities among the telephone survey, the AFQT and the 
NALS in terms of the percentages of cases that fail into each of the five levels. Table 8 confirms that there is 
little similarity. This is a point for discussion later on. 
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The demographic daia indicate that ihc less well educated, the young, nonwhites, and workers in less 
skilled occupations arc ovcr-rcprcscntcd in Level 1 in comparison to the total distribution of scores in the five 
categories. More females than males arc found in the higher levels of literacy. 

Discussion 

This research was motivated by the interest being increasingly shown by government agencies in finding 
more cost-effective ways to assess adult literacy proficiency than the use of expensive door-to-door surveying 
and testing procedures (Rcdcr, 1994; NciccAdsctt, & Rodney, 1992). Therefore, we set out to answer a fairly 
simple and straightforward question. Can we assess adult literacy proficiency by telephone? Based on the results 
reported above, we believe the answer is, yes. 

The results of the telephone and mail out surveys confirm previous research in showing strong 
relationships among listening and reading when the task demands due to differences in modalities arc kept to a 
minimum (Sticht, Hookc, & Caylor, 1981; Sinatra, 1990). For the telephone and mail out surveys, the 
correlation of .80 obtained with the Total literacy scores and the slightly lower correlations, ranging from .63 
to .73 for the separate, brief checklists, indicates that people tended to fall in similar rank order positions 
regardless of whether they had responded to the checklists by listening over the telephone or by reading them in 
written form. Therefore, by assessing people's relative knowledge by listening, we can infer quite closely the 
relative knowledge they would exhibit by reading. 
Costs of telephone vs. door-to-door surveys 

In addition to sponsoring the collection of academic skill achievement data by direct testing in the United 
Slates, the National Center for Education Statistics (NCES) also has an office for conducting telephone 
surveys. According to Dr. Kathryn Chandler of the NCES. costs per interview by telephone arc a fraction, 
certainly less than a fourth, of the costs of conducting door-to-door interviews (Chandler, 1994, personal 
communication). Chandler estimated that a sample of 6,000 telephone interviews with adults might easily be 
achieved for less than SI million. This suggests that to interview a national representative sample of 13,600 
adults, with an ovcrsampling of Black and Hispanic households, as was done in the NALS (Kirsch, Jungcblul, 
Jenkins, & Kolstad, 1993, p. 5), the cost world be around S2 million or less. This compares with the 
approximately S10.85 million cost of obtaining the national sample for the NALS. 
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Docs the telephone survey method provide as useful information as obtained with the door-to-door survey 
method? 

The differences in ihc estimates of ihc cost of telephone and door-to-door assessments, the simplicity of 
the telephone survey of literacy methodology, and the high correlation of listening with reading in the 
telephone and mail out samples, support the use of the telephone survey and checklist methodologies as a cost- 
effective way to assess adult literacy. 

However, the concept of cost-effectiveness requires that both the cost and the effectiveness of some 
alternative methods for achieving the same or at least very similar outcomes be considered. There is no 
question that the telephone survey methodology is a lower cost approach to interviewing than door-lo-door 
canvassing. But many may question whether the checklist methodology is superior to the actual performance of 
literacy tasks, as in the National Adult Literacy Survey (NALS), for usefully characterizing the literacy of the 
adult population. 

But this raises the very important question of just what it means to "usefully characterize" the literacy of 
the adult population. Why try to do this at all? Indeed, some have argued that it cannot be done. Following a 
review of six major books on adult literacy published during 1979-1990, Kazcmck (1990) expressed his 
opinion that "... attempts at defining "ability levels" and "norms" arc not only futile but potentially dangerous 
as well. How can wc possible arrive at acceptable definitions of literacy when there arc countless life goals, 
needs, and desires among the adult populations? Realistically wc cannot, but in our attempts to do so wc 
usually produce reductive lists, scales, and criteria... which arc then used to categorize large segments of the 
population, often in detrimental ways" (p. 56). 

Despite such strongly held doubts by scholars regarding the feasibility of defining and assessing adult 
literacy (sec also paper* in Vcnezky, Wagner, & Cilibcrii, 1990), the U. S. Congress passed the Adult 
Education Amendments of 1988 that required the U. S. Department of Education to submit a report to 
Congress on the definition of literacy and then to report on the nature and extent of literacy among adults in the 
nation (Campbell, Kirsch, & Kolsiad, 1992, p. 2.). With the aid of a national advisory board, the National 
Center for Education Statistics (NCES) agreed upon the definition of literacy and the three literacy scales 
discussed earlier in this paper. Then in September of 1 ( >89 the NCES awarded a four- year contract to the 



Educational Testing Service 10 conduct a nationally representative household sample survey to assess the 
literacy skills of the adult population of the United States. 

The report of the design of the NALS suited that "Results from the National Adult Literacy Survey will 
provide policymakers, business and labor leaders, educators, researchers, and citizens with vital information on 
the condition of literacy in the United States that is not currently available from the frequently administered 
school-based surveys (Campbell, Kirsch, & Kolstad, 1992, pp. 2,3,)." The report goes on to outline several 
informational products that the NALS would provide. For the present purposes, this provides a list of 
outcomes, or "benefits" that the NALS was intended to provide. This makes it possible to conduct a cost- 
effectiveness analysis to determine if the same outcomes or "benefits" as were produced by the NALS can also 
be obtained by the telephone survey approach. 

The following discussion first states the informational products that the NCES survey design report said 
the NALS would provide. This is followed by a discussion of what the NALS and the telephone survey method 
provide for the informational product under consideration. The NCES survey design report stated that the 
NALS would do the following. 

L Describe the levels of literacy demonstrated by the total adult population as well as by adults 
comprising various subgroups, including those targeted as "at risk" 

What the NALS did was develop three groups of tasks called prose, document and quantitative literacy, 
administer the tasks to samples of adults, use the tasks to scale both the adults' literacy proficiencies on each of 
the three scales and the difficulty levels of the tasks using item response theory. The difficulty level of each 
task was defined as the level of literacy needed to have an 80 percent probability of being able to perform the 
task. The NALS defined five levels of literacy proficiency using ratings of features of the tasks that included 
such things as the type and amount of inlormalion that had to be searched during the task performance 
(Moscnthal & Kirsch, 1994). On a scale ranging from 0 to 500, Level 1, the lowest level of literacy included 
tasks scaled from 0 to 225 in difficulty, Level 2 tasks ranged from 226 to 275, Level 3 tasks ranged from 276 
to to 325, Level 4 from 326 to 375, and Lcvc. 5 from 376 to 500, 

People were assigned to each level based on their proficiency scores. For instance, people designated as 
having skills in Level 1 were there because they had an KO percent probability of being able to perform the 



25 



average tasks in that level (tasks rated about 200 in difficulty). Level 2 people were those who could perform 
80 out of a hundred of the average tasks in Level 2 and so forth. Table 8 shows the percentage of the adult 
population (age 16 and older) that was placed in each of the five levels of literacy proficiency following the 
NALS procedures. 

How well do these procedures characterize the literacy skills of adults? It all seems very arbitrary. First, 
the decision to scale adults' literacy proficiency and tasks using a probability of .80 of being able to perform a 
given task is arbitrary. Why not 70 percent or even 60 percent? This would drastically change the outcomes of 
the analyses. 

Second, the decision to assign people to five levels of literacy proficiency based on their being able to 
perform 80 percent of the average tasks at a given level means that any competence to perform at higher levels 
was not "credited" to the adults. For instance, people who scored on the average in Level 1 could also perform 
fifty percent of Level 2 tasks, 25 percent of the Level 3 tasks, 20 percent of the Level 4 and one in six of the 
Level 5 tasks. This seems to leave plenty of room for uncertainty about just what the literacy levels of adults 
"really" arc. 

The telephone survey methodology also permits people to be assigned to levels, as indicated in Table 8. 
By using means and standard deviations to establish level boundaries, all the arbitrary decisions regarding the 
probability figures for task performance arc eliminated. Further, all other available test information that is 
based on normal curve statistics, such as the Armed Services Vocational Aptitude Battery, various IQ tests, the 
Scholastic Achievement Tests, etc., becomes relevant to interpreting the telephone survey data. 

2. Characterize the demonstrated literacy skills in terms of demographic and personal background 
information. 

Major demographic data collected by the NALS were also collected in the telephone survey (sec Tables 
3,4, and 5). In both assessments, literacy proficiency is positively and consistently related to education, age, 
ethnicity, income, occupational status, father and mother's education, and cxk t of engagement in literacy 
practices such as newspaper and magazine reading. These same relationships have been reported for the last 75 
years in every major mass assessment of adult "intelligence," "aptitude," or "literacy" (Sticht & Armstrong, 
1994). These types of demographic/practice data arc obtained at much less cost by the telephone survey. 
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3. Characterize the workforce oftlxe country with respect to demonstrated literacy skills and activities 
reported by individuals in various occupational categories. 

The NALS and iclcphonc surveys both characterize the literacy proficiencies and practices of adults in 
major occupational groups. Both surveys produce similar results: laborers arc not as proficient as clerical 
workers who arc not as proficient as managers and professionals. These findings have been consistently found 
in numerous adult literacy assessments since the introduction by the U. S. Army of mass testing during World 
War I (Slichi, & Armstrong, 1994). The iclcphonc survey mclh xl collects occupational da;a much less 
expensively than the door-to-door survey method. 

4. Provide an increased understanding of the skills and knowledge associated with functioning in a 
technological society. 

When the NALS research report directly raised the most important question about literacy and functioning 
in our technological society, the question that must have motivated the U. S. Congress to ask for the survey in 
the first place, and the question surely of most interest to corporate America, labor unions, adult educators, and 
adults themselves, the answer was, at best, disappointing. The report asked, "Arc the literacy skills of 
America's adults adequate? That is, arc the distributions of prose, document, and quantitative proficiency 
observed in this survey adequate to ensure individual opportunities for all adults, to increase worker 
productivity, or to strengthen America's compeililivcncss around the world (Kirsch, Jungcblut, Jenkins, & 

Kolstad, 1993, p. xviii)? " 

The NALS authors then went on to answer the question. "Because it is impossible to say precisely what 
literacy skills arc essential for individuals to succeed in this or any other society, the results of the National 
Adult Literacy Survey provide no firm answers to such questions (Kirsch, Jungcblut, Jenkins, & Kolstad, 
1993, p. xviii)." 

In short, the most important question from a policy point of view was not answered by the NALS. The 
authors went on to discuss the relationships of being in the lower literacy levels (which, it should be reiterated, 
were not so much "observed" as they were "created" by the many arbitrary decisions made in the survey study) 
to one's social standing as indicated by more limited occupational opportunities, income, and so forth. From 
this readers were invited to make inferences about how lower literacy skills may lend to limit one's functioning 
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in society. But all of these same relationships arc readily studied by the telephone survey methodology at a 
much lower cost. 

There is some reason to argue that the assessment of knowledge (by checklists and/or other methods) is a 
more useful method for characterizing the "skills and knowledge associated with functioning in a technological 
society." The NALS study in its definitions of prose, document and quantitative literacy and numerous other 
studies of reading have concluded that high levels of "prior" or "background" knowledge about what one is to 
read is a prerequisite for comprehending at a high level. Rccht & Leslie (1988) found that low reading ability 
(< 30th percentile) fifth grade studcnLs with a lot of prior knowledge about baseball were able to read, recall, 
and summari/.c information from passages about baseball better than higher ability (> 70th percentile) readers. 
In this case, high relevant knowledge offset a 30 percentile difference in general reading "skill." 

Stichtct al (1986) found lhat U. S. Navy personnel with high background knowledge about the U. S. 
Navy were able to comprehend at a 70 percent correct level with 6th grade reading skills, as measured by a 
general reading test, while personnel w'.th little background knowledge needed to be reading at the 1 1th grade 
level to achieve 70 percent comprehension. In this case, high relevant knowledge offset a five "year" difference 
in reading grade level of "skill." 

These studies of special knowledge support the idea that those with vast bodies of knowledge, perhaps 
made up of many interconnected smaller bodies of "special" knowledge, will be able to perform the broad range 
of tasks called lor in the NALS and in a literate, technological society in general better than those with more 
limited knowledge bases. Indeed, the vocabulary and other literacy checklists used in the telephone survey were 
designed to be especially sensitive to knowledge that might be gained by wide-ranging reading. That being the 
case, then such knowledge is likely to be predictive of those who will be better able to perform well on a wide 
range of literacy tasks cnountcrd in our literate society, including those of the NALS. The correlation between 
these two types of assessment* fc not known, but because the literacy checklists arc correlated with years of 
education, and years of education is correlated with performance on the NALS, then we can infer that 
performance on the checklists is correlated with the NALS, and vice versa. 

It is also likely that assessments of knowledge will better predict the performance of "real world" literacy 
tasks better than will the NALS-typc scales. For this reason, the Armed Services have spent decades and tens of 
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millions of dollars on the Armed Services Vocational Aptitude Battery (ASVAB). This lest battery consists of 
ten tests > all of which require some reading and eight of which arc tests primarily involving general and special 
vocabulary and conceptual knowledge (e.g., knowledge of geometry, electronics, automobiles, etc.) (Sticht, & 
Armstrong, 1994, pp. 31-39). These tests arc used to select applicants for military service and to predict who 
will be most likely to succeed in different kinds of technical training and jobs. This supports the position that 
knowledge assessment can serve to identify those who can use printed and written materials to function in 
society, at least in the high-technology world of the armed services. 

A final point to be made regarding the usefulness of the telephone/ knowledge checklist methodology for 
assessing adult literacy in relationship to functioning in a technological society concerns the differences 
between the findings of the NALS and the telephone survey with older adults. As discussed earlier, in the 
NALS, averaged across the prose, document, and quantitative scales, adults' literacy performance increased from 
age 16 to age 54. Then it dropped a full standard deviation for those 65 and older, and fell well below the 
levels of the youngest adults (Kirsch, Jungcblut, Jenkins, & Kolstad, 1993, p. 31). 

On the telephone/checklist survey, Total literacy scores inc r cased from age 16 to 64 and then dropped a 
half a standard deviation at age 65+ (see Table 4). However, they remained a full standard deviation above the 
scores for the young adults. This suggests that declarative knowledge may grow over most of the adult 
lifespan and remain at fairly high levels above age 65. This is consistent with a large body of research on 
"verbal intelligence" suggesting that "crystallized intelligence," i.e., breadth of declarative knowledge, grows 
and is sustained over the lifespan of adults ( Kimmcl, 1990, pp. 183-186). 

The differences between the NALS and the telephone survey for the older adulLs may reflect the differences 
between the load on working memory that the NALS tasks impose. Given the general decline in working 
memory of elderly adults, the NALS may undcrrcprcscnt their literacy competence by imposing demands on 
working memory in the context of a possibly stressful test situation that the elderly would not encounter in 
their day-to-day lives. Given the relative permanence of printed documents, older adults may be able to perform 
a much broader range of tasks in our technological society than the NALS would suggest because they can be 
selective in the tasks they encounter and they can carefully study, read and reread materials to bring their high 
levels of knowledge to bear in making sense of the materials. 



5. Interpret the findings related to information-processing skills and strategies in a way that can inform 
curriculum decisions pertaining to the education and training of adults. 

Not surprisingly, the NALS survey results and mcthdology have led some lo suggest that adult basic 
skills programs should be geared to improving adults' skills in prose, document and quantitative (PDQ) literacy 
(Moscnthal & Kirsch, 1994). Indeed, the Educational Testing Service has an interactive video, computer-based 
instructional series under development that would teach such skills. After a small pilot study with a group of 
some 10-12 adult basic skills students who worked on a brief (about 40 hours) document literacy instructional 
program, the class averaged a gain of 7.78 scale score points (about .17 standard deviations) on a NALS-like 
test of document literacy. But the adult students made three to four times as much gains on prose and 
quantitative literacy tests as on the document literacy tests. This led the instructor who administerd the pilot 
course to observe that, "The gains were interesting considering the PDQ curriculum did not include instruction 
in these skills." (Orr-Hollcy, 1992, p. 1). 

This raises questions about the validity of the three scales as distinct scales as claimed by the 
NALS/NCES developers. Others who have analyzed the NALS data have concluded that there are high 
intcrcorrclations among the three scales (around .90+) (Redcr ,1994) and that the results of factor analysis 
(Salganik & Tal, 1989) suggest that the three scales arc not really providing any different, unique information 
and that t^icy overlap considerably. The results of the pilot instructional program would seem consistent with 
this point of view. 

When asked how it was that they made such improvements in the document literacy program, some 
students commented, "it refreshed my thinking skills" ;"it made me remember some of my ways of reading and 
comprehending" ; "I got practice in taking tests by the end (of the course)"; "...all we did was read! I haven't 
read this much since I've been out of school!"; "The computer lessons made me think of questions and making 
sure I did what it asked me. That helped me read that other (prose) test better this time"; and "I think that I read 
faster now" (Orr-Hollcy, 1992, p. 1). 

From the comments, it seems that most of the gains in the document literacy pilot program were 
perceived by the students to result from re-learning or practicing what they had earlier learned to do in school. 



3 0 

33 



In other words, they seem to think that they had simply brushed up their existing competence, rather than 
having developed new competence. This type of "warm-up" effect has been found in numerous adult literacy 
programs that report making one or two "years" of gain in anywhere from 2 to 10 hours of practice (Sticht, 
1987). 

The theory behind the knowledge-based approach to literacy assessment used in the telephone survey is 
the practice-engagement theory of literacy development (Rcdcr, 1994; Stanovich, 1993). This theory holds that 
by engaging in extensive practices involving reading of a wide-ranging nature literates build vast bodies of 
knowledge (both declarative and procedural) and aulomalicity of word recognition that in turn make it possible 
to engage in and successfully complete a large number of literacy tasks. 

Consistent with practice -engagement theory, Krashcn (1993) reviewed 4 1 studies comparing free, 
voluntary reading to traditional reading instruction and reported that 16 showed free reading to be superior to 
traditional reading instruction in improving reading scores, 22 showed no differences between the two methods, 
and only 3 showed that traditional reading instruction is superior to wide-ranging reading. The longer the 
students engaged in free, voluntary reading, the more they improved in tests of reading, writing, and spelling 
over the traditional approach to reading instruction. 

This suggests one simple recommendation for curriculum development for adult literacy programs from 
the telephone/checklist mlhodology. To help people develop large bodies of knowledge and hence to become 
highly literate, literacy programs should arrange conditions that will encourage students to engage in 
extensive, wide-ranging reading over long periods of time. 

Conclusions 

From the foregoing it appears that the telephone survey method can provide all of the same five 
informational products that the NALS was designed to provide. However, the NALS sample included recent 
immigrants and longer term residents who did not speak English while the present demonstration research did 
not. That is one reason why the NALS has so many people in Level I (Tabic 8) . Many of the non-English 
speaking adults were unable to perform most of the NALS tasks and were assigned scores through estimation 
techniques that placed them in Level 1 . This procedure, could, of course, be carried out with telephone surveys. 
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One would simply assign a score of zero to non-Engiish speaking respondents and this would place them in 
Level 1 of the telephone survey. 

Alternatively, the telephone survey could be conducted in the major languages of non-English speakers, 
primarily Spanish. Because of the simplicity of the checklist knowledge assessment methodology, various 
checklists sensitive to various cultural backgrounds could be easily developed and administered to estimate 
adults' literacy in their native language. Pilot studies with Chinese immigrants showed that the longer they are 
in the west, the more likely they arc to know more of the information on the various checklists. This suggests 
that the checklist methodology might offer a good measure of acculturation. 

The NALS utilized item response theory (IRT) methods to scale tasks for difficulty. This permits the 
compilation of multiple forms of assessments with known psychometric properties without the need to 
develop new norms everytime a new assessment is needed. It also permits the determination of the probabilities 
that persons with literacy ability levels established on the same scale that the items have been scaled on will be 
able to perform each of the various items. While IRT scaling was not used in the telephone survey/checklist 
approach in this demonstration study, there is no reason in principle while this could not be accomplished. 

Finally, in regard to what these types of surveys accomplish, it should be noted from Table 8 that years 
of education alone docs not accurately portray the literacy abilities of the adult population. Accepting reports of 
years of education completed as an indicator of adult literacy understates the problem of adults with lower levels 
of literacy. Fagan (1994, p. 268) notes that surveys such as the NALS or the telephone/checklist serve the 
generally useful purpose of keeping the issue of adult literacy visible to policymakers, business and industry 
leaders, labor unions, educators and adults themselves who may be stimulated to seek additional literacy 
development. 

Perhaps the relatively low-cost of the telephone/checklist survey methodology will stimulate 
governments or businesses to monitor adult literacy development more closely and provide the resources needed 
to help more adults achieve the rewards of higher literacy. This may also be a cost-effective way to help the 
public schools be more effective. Through the intcrgcncrational transfer phenomenon, an investment in the 
education of adults may produce returns in the cducabilily of the adults' children. 
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Appendix A 



Items from the telephone survey relevant to the assessment of adult literacy. 



Item 


X 


SD 


N 


Q5A. On an average day aboul how 








many hours do you walch television? 


2.47 


1.69 


529 


Q6. On an average day aboul how 








many hours do you lisicn lo radio? 


2.68 


2.93 


533 


Q7. In an average week aboul how many 








times do you read a newspaper? 


4.40 


2.76 


534 


Q25. How oficn during an average week do you 








do each of ihe following: 








a. Read something for pleasure? 


4.68 


2.50 


528 


b. Read something because your job requires it? 


3.37 


2.80 


524 


c. Read a book to a child? 


0.64 


0.83 


529 


d. Read a book for pleasure? 


3.10 


2.78 


527 


c. Read 1 enters? 


0.91 


0.73 


527 


f. Read a newsmagazine? 


0.79 


0.67 


528 


g. Read local or national news in a newspaper? 


4.38 


2.77 


529 


h. Read ihe sports section of a newspaper? 


2.38 


2.91 


529 


i. Read the editorial section of a newspaper? 


2.69 


2.85 


529 


j. Listen to someone else in your household 








read aloud? 


0.52 


0.74 


529 


k. Read books or manuals lo help you do 








your job? 


2.34 


2.51 


523 



Author Recognition Test (ART) 

Q26. I will now read you a list of names. Some of the people in this lisi are popular writers of books, 
magazines and/or newspaper columns, and some arc not. Please just tell mc if you recognize each one as a 
writer. Please do not guess. (10 rcal/5 foils) Data arc percent of respondents saying "yes" : o each name. WSM= 
West, Stanovich, & Mitchell (1993, pp. 35-50). 











Data from WSM 




X 


SD 


N 


Reader 


Nonrcadcr 


a. Andrew Greeley 


0.26 


0.44 


498 


0.48 


0.23 


b. Irving Wallace 


0.50 


0.50 


501 


0.69 


0.33 


c. Nancy Roscr (foil) 


0.05 


0.22 


490 


0.02 


0.00 


d. James Clavcll 


0.44 


0.50 


502 


0.65 


0.32 


c. Isabel Beck (foil) 


0.14 


0.35 


494 


0.01 


0.03 


f. Louis L'Amour 


0.64 


0.48 


515 


0.69 


0.43 


g. Robert Ticrncy (foil) 


0.20 


0.40 


491 


0.02 


0.03 


h. Judith Krantz 


0.58 


0.49 


509 


0.77 


0.48 


i. F.E. Bryant (foil) 


0.13 


0.34 


494 


0.00 


0.02 


j. James Michcncr 


0.66 


0.47 


516 


0.90 


0.65 


k. Sidney Sheldon 


0.82 


0.39 


517 


0.88 


0.75 


1. Gerald Duffy (foil) 


0.09 


0.29 


494 


0.01 


0.03 


m. J.R.R. Tolkien 


0.58 


0.49 


506 


0.80 


0.57 


n. Joseph Wambaugh 


0,49 


0.50 


512 


0.63 


0.29 


o. Bob Woodward 


0.43 


0.50 


502 


0.73 


0.36 



ERIC 



3 6 



3G 



BEST COPY AVAILABLE 



Magazine Recognition Test (MRT) 

Q27. 1 will now ^d you ^ list of magazine names. Some of the names arc real magazines, and some are 
not. Please listen to the name* ..id tell mc if you recognize each as an actual magazine. Please do not guess. 
(9 real/6 foils). WSM= West, Stanovich, & Mitchell (1993, pp. 35-50). 











Data from WSM 




X 


S D 


N 


Reader 


Nonreader 


a. Harper's Magazine 


0.73 


0.45 


517 


0.77 


0.52 


b. Gentlemen's Quarterly 


0.66 


0.47 


515 


0.73 


0.53 


c. Fitness Today (foilj 


0.46 


0.50 


489 


0.26 


0.36 


d. American Journal Review (foil) 


0.35 


0.48 


486 


0,04 


0.08 


e. New Yorker 


0.79 


0.41 


517 


0.89 


0.69 


f. Trends American (foil) 


0.10 


0.31 


478 


0.00 


0.02 


g. Ladies Home Journal 


0.81 


0.39 


516 


0.92 


0.79 


h. Scientific American 


0.53 


0.50 


505 


0.61 


0.43 


i. Town & Country 


0.75 


0.43 


508 


0.72 


0.54 


j. Health & Life (foil) 


0.22 


0.42 


483 


0.09 


0.15 


k. Psychology Today 


0.72 


0.45 


513 


0.83 


0.67 


I. Outdoor Times (foil) 


0.27 


0.45 


484 


().'05 


0.09 


m. Esquire 


0.90 


0.31 


520 


0.98 


0.83 


n. Forbes 


0.81 


0.39 


520 


0.95 


0.81 


o. Motor Sports (foil) 


0.24 


0.43 


481 


0.08 


0.12 



Q28. Many people tell us that they have difficulties in understanding what they read. In general, would 
you say that you sometimes have trouble understanding what you read (1 point), you understand most of what 
you read (2 points), you understand nearly all you read (3 points), or you understand all you read (4 points) ? 
(Don't know/Nol applicable (code as 9). 

Frequency (%) 
12 3 4 

Average rating: 2.95 0.88 522 6.7 21.5 41.8 30.1 

Q29. How connected would you say reading ability is to how well you do your job at work? Net at all 
connected (1), not very connected (2), somewhat connected (3), very connected (4). (Don't know/Not applicable 
(code as 9). 

Frequency (%) 
12 3 4 



Average rating: 



3.53 0.80 4.71 4.5 



6.2 



21.4 67.9 



Q30. For each of the following statements, just tell mc whether you agree strongly (1 point), agree (2 
points), disagree (3 points), disagree strongly (4 points). (Don't know/Not applicable (code as 9). 



a. If I understood what I read better, I 
would gel more respect where I work. 

b. If I understood what I read better, I 
would be able to do better at the job. 

c. Il l understood what 1 read better, 1 
would be able to earn more money. 

d. I can understand what I read well 
enough to do my job well even when 
there is a lot of distraction. 

c. I can understand what I read well 
enough to do my job well even when 
I am under a lot of pressure to meet 
a deadline. 









1 


Frequency (%) 
2 3 


4 


2.51 


0.88 


452 


14.4 


31.6 


42.3 


11.7 


2.36 


0.9 1 


454 


19.4 


15.7 


34.4 


10.6 


2.50 


0.83 


457 


13.1 


32.4 


45.7 


8.8 


1.82 


0.61 


467 


29.1 


60.4 


10.1 


0.4 


1.80 


0.60 


468 


29.5 


61.1 


9.2 


0.2 
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Cultural Literacy Test 

Q31. 1 will now read you a list of names of persons. Some of the people in this list arc popular famous 
persons, and some arc not. Please listen to each name and tell rnc if you know the person to be famous. Do 
not guess. (17 real/6 foils). WSM= West, Sianovich, & Mitchell (1993, pp. 35-50). 

Data from WSM 





X 


SD 


N 


Reader 


Nonreadcr 


a. Ingmar Bergman 


0.73 


0.44 


511 


0.95 


0.78 


b. John Gottman (foil) 


0.11 


0.31 


485 


0.02 


0.03 


c. Steve Biko 


C 12 


0.32 


485 


0.46 


0.23 


d. Harry Houdini 


0.90 


0.30 


514 


0.93 


0.75 


e. Paul Cezanne 


0.41 


0.49 


497 


0.64 


0.26 


f. Marie Curie 


0.71 


0.45 


509 


0.88 


0.65 


g. DaleBlyth (foil) 


0.05 


0.23 


479 


0.02 


0.01 


h. Jean Jacques Rouscau 


0.57 


0.50 


505 


0.51 


0.34 


i. Enrico Fermi 


0.30 


0.46 


493 


0.55 


0.26 


j. Darwin Muir (foil) 


0.15 


0.35 


483 


0.01 


0.00 


k. Carlos Fucntcs 


0.33 


0.47 


493 


0.31 


0.15 


1. George Gershwin 


0.84 


0.37 


510 


0.98 


0.82 


m. Rcinhold Klicgcr (foil) 


0.09 


0.29 


482 


0.02 


0.01 


n. Rosa Parks 


0.56 


0.50 


500 


0.57 


0.35 


o. Margaret Mead 


0.69 


0.46 


508 


0.87 


0.64 


p. Miriam Sexton (foil) 


0.12 


0.32 


482 


0.02 


0.02 


q. Georgia O'Kecfc 


0.46 


0.50 


496 


0.64 


0.43 


r. Sylvia Plath 


0.25 


0.44 


488 


0.39 


0.13 


s. Cole Porter 


0.73 


0.44 


509 


0.92 


0.69 


t. Walter Raleigh 


0.65 


0.48 


505 


0.76 


0.54 


u. W. Patrick Dickson (foil) 


0.10 


0.30 


484 


0.00 


0.02 


v. Margaret Sanger 


0.37 


0.48 


493 


0.43 


0.22 


w. Greta Garbo 


0.90 


0.30 


516 


1 .00 


0.88 



Q. 33. In general, how frequently do famil>' members or friends help you with the following 
activities? Never (0), annually (1), monthly (2), weekly (3), daily (4). (Don't know/Not applicable 
(code as 9). 













Frequency (%) 














0 


1 


2 


3 


4 


a. Filling out forms 


0.58 


0.91 


517 


64.4 


19.9 


10.3 


4.4 


1.0 


b. Reading or explaining 


















newspaper articles or other 


0.52 


0.95 


521 


72.0 


11.9 


8.8 


6.7 


0.6 


written information 


















c. Dealing with government 


0.70 


0.94 


514 


56.6 


23.3 


15.0 


3.9 


1.2 


agencies, public companies, 


















business, medical personnel, 


















etc. 


















d. Writing notes and letters 


0.44 


0.96 


518 


79.0 


7.1 


6.6 


5.6 


1.7 


c. Helping you with things 


















you need to read at work 


0.29 


0.77 


479 


85.0 


6.1 


5.2 


2.7 


1.0 
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Vocabulary Literacy Test (VLT) 

Q36. 1 will now read you a list of vocabulary words. Some of the words in this list are real words, and 
some arc not. Please listen to the words and tell mc if you know the word to be real. Please do not cucss (14 
rcal/7 foils). WSM= West, Stanovich, & Mitchell (1993, pp. 35-50). 

Data from WSM 
Reader Nonreadcr 
Not reported in WSM 





X 


SD 


N 


a. absolution 


0.77 


0.42 


507 


b. arratc (foil) 


0.30 


0.46 


496 


c. nitrous 


0.71 


0.45 


512 


d. audible 


0.90 


0.30 


513 


e. ceiloplaty (foil) 


0.06 


0.24 


476 


f. comcctial (foil) 


0.05 


0.22 


472 


g. confluence 


0.53 


0.50 


494 


h. connote 


0.53 


0.50 


498 


i. polarily 


0.84 


0.37 


516 


j. disconcert 


0.80 


0.40 


506 


k. inelniy (foil) 


0.18 


0.38 


483 


1. nuance 


0.74 


0.44 


510 


m. irksome 


0.65 


0.^8 


503 


n. ubiquitous 


0.61 


0.49 


504 


o. mctention (foil) 


0.09 


0.28 


481 


p. ncotalin (foil) 


0.11 


0.31 


481 


q. purview 


0.44 


0.50 


490 


r. nonquasity (foil) 


0.13 


0.33 


481 


s. optimize 


0.85 


0.35 


513 


t. eventuate 


0.48 


0.50 


491 


u. epicurean 


0.57 


0.50 


500 



Q40. In terms of what is required for your occupation, how would you rate your own reading skills 9 Not 
at all adequate (1), not as good as should be (2), adequate for job (3), more than adequate (4). (Don't ^now/Not 
applicable (code as 9)). 



Average rating: 



Frequency (%) 
12 3 4 

3.52 0.58 499 0.4 3.0 41.1 55.5 



Q45. Do you think you could get a (better) job if you received additional training in reading or writine 
English? (l=ycs: 2=no) (9=Don't know/Not applicable). 



Average rating: 



1.67 0.47 483 



Frequency {%) 
J I 

33.3 66.7 



Q46. Do you think you could get a (better) job if you received additional training in mathematics 9 
(l=ycs: 2=no) (9=Don't know/Not applicable). 

Frequency (9r) 
J 2 

Average rating: 1 .62 0.49 484 37.8 62.2 
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Tabic 1. Comparison of the San Diego telephone and mail 
out samples with the 1990 U. S. Census figures for San 
Diego County. 



San Diego Survey Sample U. S. 
,Variat>lcs a _ Telephone Mail Qui Census 



Marital Status 


(519) 


(137) 




Single 


30.4 


24.8 


30.2 


Married 


52.0 


54.7 


51.8 


Separated/Other 


2.0 


1.6 


2.5 


W lUUWWJ 


o.z 


O f\ 

8.0 


5.6 


Divorced 


9.4 


10.9 


10.0 


Household Income 


(478) 


(131) 




Under S 10,000 


9.0 


6.1 


6.8 


$10,000-49,999 


59.6 


59.5 


56.5 


Over S50.(X)0 


31.4 


34.4 


36.7 


Age 


(512) 


(137) 




18-24 


15.4 


12.4 


17.8 


25-34 


23.6 


2 1 ^ 


26.4 


35-44 


27.7 


26.3 


20.1 


45-59 


17.8 


23.4 


16.4 


60-64 


3.3 


2 2 


4.7 


65+ 


12.1 






Education^ 


(517) 


(135) 




0-11/Noi HS 


4.8 


2.2 


18. 1 


12/Complctcd HS 


20.7 


20.7 


22.8 


13-15/Somc Coll. 


35.4 


33.3 


25.6 


16/AAor BA 


18.0 


17.0 


24.7 


1 7+/Morc 


21.1 


26 7 


8.8 


Race/Ethnicity 


(520) 


(137) 




White 


72.9 


82.5 


65.6 


Black 


4.6 


3.6 


6.0 


Asian 


5.4 


5.1 


7.5 


Other 


3.5 


1.5 


0.0 


Hispanic-All 


13.7 


7.3 


20.0 


Gender 


(522) 


(138) 




Male 


47.9 


44.2 


50.9 


Female 


52.1 


55.8 


49.1 



a Numbers arc percentages of the samples and 1990 U. S. 
Census with the characteristics listed. Education categories 
are codified differently for sample and census data, with 
sample data matched as closely as possible to census 
categories. Labels for sample data precede labels for census 
data for education. Numbers in brackets arc total numbers 
for samples for a given characteristic 
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Tabic 2. Correlations of checklist literacy scores on the telephone survey (listening) 
with scores on the mail out (reading) survey. 









Reading 








Listening 


1 


2 


3 


4 


5 


6 


1. Total Literacy 


M 


.71 


.64 


.74 


.62 


.38 


2. ART 


.67 


21 


.50 


.60 


.43 


.29 


3. MRT 


.63 


.51 


M 


.57 


.39 


.31 


4. CLT 


.71 


.59 


.55 


Jl 


.53 


.36 


5. VLT 


.59 


.51 


.35 


.51 


,63 


.30 


6. Cloze 


.39 


.34 


.41 


.34 


.31 


1.00 



xat^t x m • >um,lKU uvu luurcnccKiisis; AK I =Aulhor Recognition Tcni- 

MRT=Maga Z .nc Recognition Test; CLT= Cultural Literacy Test; Vlkv3ui7v 
«SJS L Signiricanl: a " olh ™ significant beyond p < .01. Underlined r's 

arc alternate modality, tcsi-reicsi rchability coefficients. N's ranged from 135-137 except 
for the cloze test where N's ranged from 1 08- 1 09. P 
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Tabic 3. Correlations among literacy and demographic variables. Correlations below the diagonal are from West, Slanovich 



& Mitchell (1993) using versions of the checklists with more 


items. 














Variables 


1 


t 


3 


4 


5 


6 


7 


8 


9 


10 


11 


1. Education 


\1.00 


.07* 


.23 


.31 


.31 


.36 


.37 


.34 


.14 


-.10 a 


.34 


2. Age 


.16^ 


si .00 


.25 


.16 


.25 


.19 


.27 


.08* 


.30 


.05* 


.04* 


3. ART 


.33 


.16N 


si .00 


.61 


.61 


.53 




.23 


.27 


. .10 a 


.23 


4. MRT 


.33 


.06 


.34\ 


J .00 


.58 


.54 


M 


.22 


.30 


.05* 


.23 


5. CLT 


.48 


.21 


.43 


.60*^ 


1.00 


.62 


M 


.23 


.32 


.05* 


.18 


6. VLT 


.53 


.26 


.46 


.48 


,7l\ 


,1.00 


31 


.25 


.29 


-.04* 


.26 


7. Total Literacy 














si .00 


.28 


.36 


.05* 


.27 


8. Practice 
















sl.00 


.09 a 


-.16 


.26 


9. Ethnicity 


















sJ.OO 


.06* 


.13 


10. Gender 




















sJ.OO 


-.13 


1 1 . Annual Income 






















. 1.00 



ART=Aulhor Recognition Test: MRT=Maga/.inc Recognition lust; CLT= Cultural Literacy Test; VLT=v OC abuIary 
Literacy Test. Total Literacy^ scores summed over the four checklists; Practice = mean scores on questions 
25a through k (sec appendix) for different reading practices. *= Not significant, a= p < .()5, all others significant 
beyond p < .01 . Underlined r's are part-whole correlations. Ethnicity= nonwhitcs (0) and whites ( 1 )• 
gcndcr= males (1) and females (2). 
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Tabic 4. Scores on Total Literacy by demographic groups. 



Education 


X 


SD 


N 


Age 


X 


SD 


N 


Occupation 


X 


SD 


N 


0-8 years 


38.6 


29.9 


6 


16-18 


24.4 


18.4 


14 


Laborers 


28.1 


22.7 


50 


9-12 


32.3 


22.3 


126 


19-24 


30.2 


22.5 


65 


Semi/Ski lied* 


40.7 


22.8 


103 


13-14 


42.0 


22.7 


146 


25-39 


41.7 


23.0 


198 


Clerical/Sales 


42.9 


24.8 


97 


15-16 


51.1 


23.0 


130 


40-54 


57.2 


23.8 


141 


Tcch./Engrs* 


52.4 


23.1 


60 


17+ 


59.9 


23.1 


109 


55-64 


57.3 


22.5 


32 


Man-Exc-Prf* 


54.6 


23.3 


165 










65 + 


46.1 


23.0 


62 










Semi-skilled & skilled; technical and engineers; managers, executives and professionals. 








Education 


X 


SD 


N 


Elhniciiv 


X 


SD 


N 


Falhe 's Educ. 


X 


SD 


N 


< S10K* 


34.6 


25.5 


43 


Caucasian 


51 2 


22.8 


379 


0-8 years 


45.6 


25.5 


84 


10-19K 


40.7 


26.6 


54 


Black 


34.2 


19.3 


24 


9-12 


44.6 


25.0 


182 


20-29K 


38.1 


24.3 


98 


Hispanic 


29.8 


24.3 


71 


13-14 


47.2 


21.0 


44 


30-39K 


46.6 


23.3 


69 


Asian 


28.9 


27.9 


78 


15-16 


48.6 


23.2 


84 


40-49K 


49.4 


21.4 


64 


Other 


3.3.3 


22.7 


18 


17 + 


52.9 


24.4 


72 


50+K 


52.5 


23.9 


150 


















*K-thousand 
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. Average frequency of engaging in various literacy practices in a week by demographic 



Education 


X 


$D 


N 


Ape 


X 


SD 


N 


Occupation 


X 


$D 


N 


0-8 years 


1.8 


0.7 


6 


16-18 


2.1 


1.1 


14 


Laborers 


2.4 


1.3 


51 


9-12 


2.3 


1.2 


127 


19-24 


2.2 


1.1 


66 


Semi/Skilled* 


2.6 


1.1 


103 


13-14 


2.6 


1.1 


147 


25-39 


2.8 


1.2 


198 


Clerical/Sales 


2.5 


1.2 


98 


15-16 


2.9 


1.1 


131 


40-54 


3.1 


1.2 


142 


Tech./Engrs* 


3.1 


1.0 


60 


17+ 


3.3 


1.2 


109 


55-64 


2.7 


1.2 


33 


Man-Exc-Prf* 


3.1 


1.2 


165 • 










65+ 


2.6 


1.2 


62 










•Semi-skilled & skilled; technical and engineers; managers, executives and professionals. 








Education 


X 


SD 


N 


Ethnicity 


X 


SD 


N 


Father's Eriur. 


X 


SD 


N 


< SI OK* 


2.2 


1.4 


43 


Caucasian 


2.8 


1.2 


380 


0-8 years 


2.7 


1.1 


84 


10-19K 


2.5 


1.2 


55 


Black 


2.7 


1.0 


25 


?-u 


2.8 


1.2 


183 


20-29K 


2.5 


1.1 


98 


Hispanic- 


2.5 


1.2 


72 


13-14 


2.7 


1.2 


45 


30-39K 


2.8 


1.2 


69 


Asian 


2.6 


1.1 


28 


15-16 


2.9 


1.2 


84 


40-49K 


2.8 


1.0 


65 


Other 


2.4 


1.2 


18 


17+ 


3.0 


1.2 


72 


50+ K 


3.1 


1.2 


150 


















*K-thousand 
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Tabic 6. Component loadings for practice items (questions 25a-k in appendix) after 
varimax rotation (n=538). 

Practice Factors 

12 3 4 



Times Per Week 


News 


Job 


Pleasure 


Family 


Q25G 


Read local or national news 


.84 








Q25I 


Read editorial section 


.79 








Q25H 


Read sports section 


.74 








Q25F 


Read newsmagazines 


.43 








Q25K 


Read books or manuals on job 




.88 






Q25B 


Read because job requires it 




.86 






Q25E 


Read letters 




.31 






Q25D 


Read a book for pleasure 






.88 




Q25A 


Read for pleasure 






.86 




Q25J 


Listen to others read 








.79 


025C 


Read a book to a child 








.79 


Dashes represent component loadings lower than .30. 



Tabic 7. Correlations among demographic, litciacy knowledge, and the four literacy 
practices factors of tabic 6 (n=538). 

Practice Factors 

12 3 4 
Variables News Job Pleasure Family 



Education 


.17 


.32 


.22 


.02 a 


Age 


.26 


-.19 


.15 


-.19 


Gender* 5 


-.23 


-.18 


.11 


.13 


Ethnicity c 


.08 a 


-.oi a 


.15 


-.08 a 


Income 


.19 


.18 


.08 a 


.05 a 


Father's education 


-.09 


.20 


.()6 a 


.()3 a 


Mother's education 


-.03 a 


.13 


.06" 


.06 a 


Latino d 


.08 a 


.04 a 


.08 a 


-.10 


Total literacy 


.17 


.12 


.30 


-.13 


ART 


.12 


.09 


.28 


-.13 


MRT 


.10 


.10 


.24 


-.07 a 


CLT 


.15 


.07 a 


.25 


-.08 a 


VLT 


.14 


.17 


.25 


-.17 



a not significant at p < .05. "gender: males 1 1 fcmalc=2. c Ethnicity: whitcs=l; 
nonwhitcs =0. ^Latino: Arc you of Mexican, Hispanic, or \atino descent? ycs=l, 
no=2. Total Literacy= scores summed over the four checklists; ART=Author 
Recognition Test; MRT=Magazinc Recognition Test: CLT= Cultural Literacy 
Test; VLT=Vocabulary Literacy Test. 
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Table 8. Percentage of telephone survey respondents falling into 
each of five levels of literacy on the Total Literacy scale. 

Literacy Levels* 



Variables 


N 


1 


2 


3 


4 


5 


Total 


538 


19.2 


14.1 


31.4 


16.7 


18.6 


Normal Curve 3 




16.0 


15.0 


38.0 


15.0 


16.0 


AFQT Categories" 


7.0 


24.0 


32.0 


33.0 


4.0 


NALS Levels'-' 




22.0 


27.0 


31.0 


17.0 


3.0 


Education 














0-8 


6 


33.3 


16.7 


16. -7 


0.0 


33.3 


9-12 


126 


31.0 


30.2 


21.4 


11.9 


5.6 


13-14 


146 


22.6 


22.6 


21.9 


18.5 


14.4 


15-16 


130 


13.1 


13.1 


25.4 


26.9 


21.5 


17* 


109 


9.2 


11.0 


12.8 


23.9 


43.1 



Age 



16-18 


14 


35.7 


42.9 


21.4 


0.0 


0.0 


19-24 


65 


41.5 


23.1 


16.9 


13.8 


4.6 


25-39 


198 


21.7 


21.2 


23.2 


21.7 


12.1 


40-54 


141 


9.9 


13.5 


15.6 


24.1 


36.9 


55-64 


32 


6.3 


12.5 


28.1 


18.8 


34.4 


65+ 


62 


17.7 


19.4 


22.6 


21.0 


19.4 



Ethnicity 
White 
Black 
Hispanic 
Asian 
Other 



379 
24 
71 
28 
18 



Occupation 

Laborer 50 
Semi/Skill 103 
CIcrk/Salcs 97 
Tech/Engr 60 
Mn/Ex/Prf 165 



Gender 
Male 
Female 



250 
272 



11.9 

25.0 
45.1 
42.9 
33.1 



42.0 
21.4 
21.6 
10.0 
12.1 



2i o 



17.4 
37.5 
19.7 
21.4 
38.9 



26.0 
22.3 
23.7 
11.7 
15.8 



19.2 
19.5 



22.4 
16.7 
18.3 
14.3 
5.6 



14.0 
25.2 
18.6 
31.7 
15.2 



22.0 
iiL 



23.2 
20.8 
8.5 
14.3 
11.1 



12.0 
20.4 
16.5 
23.3 
25.5 



18.8 
21.3 



25.1 

8.5 
7.1 
11.1 



6.0 
10.7 
19.6 
23.3 
31.5 



18.8 
21.3 



* Literacy levels arc based on the Total Literacy score (sum of the 
four checklists). Range of percent correct scores for each level arc- 
Level 1=0-20, Level 2=21-32, Level 3=33-58, Level 4-59-70, and 
Level 5=71 -UK). a Normal Curvc=thc percentage of cases under the 
same areas of the normal curve that were used to define the five 
literacy levels. b AFQT=Armcd Forces Qualification Test categories 
numbered from 1 (low) to 5 (high) instead of 5 (low) to 1 (high) 
as the Services do. c NALS=National Adult Literacy Survey. 




